Artificial intelligence methodology to automatically generate interactive play along songs

ABSTRACT

A method and system of using Artificial Intelligence to automatically create and generate an interactive play-along song from a selected audio file/song imported from either a music streaming service, a personal music library and the like, that can later be played on an interactive music engine.

PRIORITY

This application is a continuation of U.S. patent application U.S. Ser.No. 16/036,352, now issued U.S. Pat. No. 10,534,811, filed Jan. 29, 2018entitled ARTIFICIAL INTELLIGENCE METHODOLOGY TO AUTOMATICALLY GENERATEINTERACTIVE PLAY ALONG SONGS, the teachings of which are incorporatedherein in their entirety.

TECHNICAL FIELD

The present disclosure relates to the use of Artificial Intelligence toautomatically create and generate a multi instrument/sound/noteinteractive play-along song from a selected audio file/song importedfrom either a music streaming service, a personal music library and thelike.

BACKGROUND

Interactive play-along songs can be played by a multi-media platform,such as available from Beamz Interactive, Inc. of Scottsdale Ariz. Sucha platform is described in commonly assigned U.S. Pat. Nos. 7,504,577,7,858,870, 9,542,919 and 9,646,588, the teachings of each which areincorporated herein by reference in their entirety.

The Beamz Cross Platform (i.e. IOS, Android, PC, MAC, AR, MR, VR, etc.)Interactive Music Engine is an award winning rich multi-media technologyand platform configured to allow a user to selectively trigger varioustriggers each associated with a subset of a musical composition, to bothcompose and play sympathetic music. Different subsets/Midi files of amusical composition can be selectively assigned to the triggers. Theplatform includes triggers, and an electronic processor generating acontrol signal as a function of the triggers selected by a user. Theelectronic processor also generates the control signal as a function ofa plurality of music programs. Each music program comprises soundelements comprising a subset of a musical composition, and the musicprograms are correlated to each other. The electronic processorgenerates audio signals indicative of audible musical sounds as afunction of the control signal.

There is a need to use Artificial Intelligence to automatically createand generate a multi instrument/sound/note interactive play-along songfrom a selected audio file/song imported from either a music streamingservice, a personal music library and the like, that can later be playedon an interactive music engine.

SUMMARY

This disclosure comprises a method and system of using ArtificialIntelligence to automatically create and generate an interactiveplay-along song from a selected audio file/song imported from either amusic streaming service, a personal music library and the like, that canlater be played on an interactive music engine.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are included to provide a furtherunderstanding of the invention and are incorporated in and constitute apart of this specification, illustrate embodiments of the disclosure andtogether with the description serve to explain the principles of thedisclosure.

FIG. 1 illustrates a diagram of a system for multilayered mediaplayback;

FIG. 2 illustrates the Standard Midi numbering scheme used by anapplication engine;

FIG. 3 illustrates a Virtual Song map prepared by the applicationengine;

FIG. 4 illustrates the Scale based templates that supply generic notesfor further transpositions, and illustrates two possible Scale NoteTemplates;

FIG. 5 illustrates how a Global Key Offset transposes the preliminaryScale Note pool to become a Seed Note Pool in the scanned song (root)key;

FIG. 6 illustrates how Global Regional Offsets are used to dynamicallytranspose Seed Notes to accommodate key changes during song play;

FIG. 7 illustrates two selectable Instruments and their definitions;

FIG. 8 illustrates how the preliminary seed note pool is expanded by theapplication engine to fill an Instrument's specific Octave Range;

FIG. 9 illustrates how Offsets are dynamically applied by applicationengine 1007 to an Instrument's preliminary seed note pool to produce aplayable sympathetic note pool in real time;

FIG. 10 shows an example of a 13 step Melody Pattern that will play“Mary Had a Little Lamb” melody in any Key and on any Instrument; and

FIG. 11 illustrates the main processing flow executable by theapplication engine configured to generate an interactive play-along songaccording to this disclosure.

DETAILED DESCRIPTION

Interactive Music Engine

This disclosure is directed to using Artificial Intelligence toautomatically create and generate an interactive play-along song from aselected audio file/song imported from either a music streaming service,a personal music library and the like. The generated interactiveplay-along song can be played by a multi-media platform, such asavailable from Beamz Interactive, Inc. of Scottsdale Ariz. Such aplatform is described in commonly assigned U.S. Pat. Nos. 7,504,577,7,858,870, 9,542,919 and 9,646,588, the teachings of which areincorporated herein by reference in their entirety. These methods andillustrations show interactive music content being generated for directuse with the Beamz Interactive Music Engine. However, all songproperties and generated note pools can be used by other music enginesas long as they can accommodate Tempo and Key changes throughout a song;and they can use pools of sympathetic notes that are appropriate to thekey changes as the song plays.

The Beamz Interactive Music Engine includes an electronic processorgenerating a control signal as a function of trigger objects selected bya user, the electronic processor also generating the control signal as afunction of a plurality of music programs, wherein each said musicprogram comprises sound elements comprising a subset of a musicalcomposition, and the music programs are correlated to each other, theelectronic processor generating an audio signals indicative of audiblemusical sounds as a function of the control signal.

FIG. 1 illustrates a diagram of a system 1000 for multilayered mediaplayback.

The system 1000 can be implemented in electronic device 1001, andembodied as one of a computer, a smart phone, a tablet, a touchscreencomputer, and the like having a display 1004.

Application engine 1007 is operable on an electronic processor 1006 andreceives one or more inputs 1005 from the multiple trigger objects 1003within the application environment 1001.

Application engine 1007 controls playback of media files 1011 that arecombined to form a multilayered media file based on one or more oftrigger inputs 1005, and definition file 1010 via sound engine 1008. Themedia files 1011 can be one or more MIDI files, samples such as .wav and.mp3 files, video files in a plurality of formats, and/or any otheraudio or video file format.

With a short trigger break, the trigger 1003 is held broken for asubstantially short period of time, such as with a threshold for theshort period of time of 0.5 seconds or less. The application engine 1007can use the short break to trigger a one-shot, play a single note in astreamed sequence, or start and stop a loop.

With a long trigger break, the trigger 1003 is held broken for a longerperiod of time, such as with a threshold for the longer period of timeof 0.5 seconds or more. Additional thresholds may be used for longtrigger breaks with each threshold associated with a different action tobe taken by the application engine 1007.

The application engine 1007 can use a long trigger break to Pulse(stream) notes.

Processor 1006 is configured such that visual outputs from applicationengine 1007 are displayed on display 1004 and output from sound engine1008 is played on speaker 1009.

The combination of application engine 1007 and sound engine 1008 form anapplication on the processor 1006. The processor 1006 is configured toselectively associate a music programs with each of the plurality oftrigger objects. The processor 1006 is configured such that when one ofthe trigger objects 1003 is in a first state for a prolonged period oftime successive said audible musical sounds are generated, such that,for instance, the musical program associated with the trigger objectcontinues to play uninterrupted, along with any other music programsthat are playing in response the associated trigger object beingtriggered.

Display 1004 displays the application environment 1002.

When a trigger object 1003 is triggered by a user, trigger-specificvisual output from application engine 1007 can be shown to alter thedisplay properties or attributes of any element in the display 1004.

FIG. 2 illustrates the Standard Midi numbering scheme used byapplication engine 1007 according to the methods of this disclosure. Allillustrations & examples herein are based on the Standard Midi NoteNumbering scheme.

Midi is based on a chromatic music scale that breaks an octave into 12notes (semitones) shown at 11. There are 128 possible Midi notes 11spanning 10 octaves, shown at 12.

The lowest possible Midi note 11 is “C” in Octave #0 shown at 101, thehighest Midi note 11 is “G” in Octave #10 shown at 102.

As illustrated in FIG. 2, each Midi note 11 is identified by its Keyfollowed by the Octave 12 in which the Key falls. For example, Midi noteC5 represents Middle “C” in Octave #5 shown at 103.

All Midi notes 11 are numbered at 13 sequentially from the lowest to thehighest (0-127)

By using a combination of generic Offset Templates, this standard Midinote numbering scheme is used as a foundation for numerically generatingplay-along musical notes that will always be sympathetic or compatiblewith the analyzed audio file/song in real time.

FIG. 3 illustrates a Virtual Song map 20. This Virtual Song map 20 is agraphic (Music Staff) illustration that is prepared by applicationengine 1007 executing on processor 1006 from scanning an imported audiofile/song file according to this disclosure.

When the song is played by application engine 1007 and song engine 1008,it follows this Virtual Song map 20, responding to the markers as theyare encountered.

This example shows the data derived by application engine 1007 fromperforming Tempo & Chromatic analysis on a sample 10 bar (audio) audiofile/song that was recorded in the key of C major at an approximatetempo of 120 BPM.

The application engine 1007 performs Tempo Analysis on the audiofile/song to place Tempo value markers 21 in the Virtual Song map 20 toestablish or change the tempo at that point in the audio file/song. Whena Tempo value marker 21 is encountered during playback, the currentTempo is set to the value contained in the Tempo value marker 21.

The application engine 1007 performs Chromatic Analysis to add RegionalKey markers 22 to the Virtual Song map 20 to indicate chromatic changesand their location within the audio file/song.

Each Regional Key marker 22 contains the chromatic global Region KeyOffset 23 needed to transpose Seed Notes 42 from the detected Root Key26 to Region specific Seed Notes 52 in the new key.

For example, if the audio file/song is detected to be in C, and the scandetects that, during play, the key changes to F, a Regional Key marker22 is placed at that location in the Virtual Song map 20 with an offsetvalue 23 of +5.

When a Regional Key marker 22 is encountered during playback of theaudio file/song, the global Region Key Offset 23 is set to the valuecontained in the marker 22.

The application engine 1007 creates Song Sections (Chromatic Regions)shown at 28. Each chromatic region 28 becomes a Song Section (subset ofsong) in the song's definition as a whole and it marks a key change inthe audio file/song.

In this example, there are 6 Song Sections 28—each having their ownproperties. Key 22, Regional (key) offset 23, and Length 25.

The combined lengths 25 of all the Song Sections 28 constitute thesong's main loop length 29.

Use of Transposition Offsets

The following methods for generating sympathetic note pools are based onthe use and application of multiple “Offsets”.

All generated play-along notes begin from preprogrammed scale noteswhich are generated from standard templates, in a standard key, within astandard octave.

The offsets described herein are used to transpose these notes to adifferent key as required by the audio file/song.

Offsets are expressed as semitone values—the number of semitones a given(root) note needs to be transposed into the target note.

For the sake of simplicity, all illustrations and methods describedherein use the standard MIDI note numbering scheme to easily show hownotes can be transposed by semitones.

However, these transposition offset methods apply to both MIDI notes andnote samples as well.

With root notes in MIDI format, the offset amount is added to the MIDInote number for the root note to obtain the MIDI note number of thetarget note.

With sampled notes, audio transposition software is used to apply anoffset value to transpose a sampled root note to a sample in thefrequency of the target key.

FIG. 4 illustrates the Scale based templates that supply generic notesfor further transpositions, and illustrates two possible Scale NoteTemplates.

After chromatic scanning is completed by application engine 1007, thescanned key data is used to select an appropriate seed note template 31for the scanned audio file/song. A Global Root Key Offset 24 (FIG. 5) isestablished that, when applied, transposes the templated scale note pool32 from the key of C to the detected key for the song.

All Note Templates are based on the standard Midi note numbering scheme(FIG. 2) and are prepared in the key of C with a Root Note of CO 101(lowest possible Midi note).

Scale note templates supply the preliminary note pool 32 that ismusically sympathetic to the overall scanned audio file/song. Theirassociated Midi note numbers 33 are used to mathematically transposethem to the proper key and octave as needed.

FIG. 5 illustrates how a Global Key Offset transposes the preliminaryScale Note pool 32 to become a Seed Note Pool 42 in the scanned song(root) key.

After chromatic scanning is complete, the scanned key data is used todetermine a Global Root Key Offset 24 which is applied to transpose thepreliminary sympathetic note pool 32 from the key of C to the detectedkey for the audio file/song.

In this illustration, the chromatic scanning determined the audiofile/song to be in the key of G Major and the Global Root Key Offset 24is set to +7. Adding this offset to the Midi note numbers from the CMajor template 33 transposes the seed note pool to the key of G Major42, 43.

FOR THE SAKE OF SIMPLICITY, ALL OTHER FIGURES ARE ILLUSTRATED WITH ANEXAMPLE SONG IN THE KEY OF C. (GLOBAL KEY OFFSET=0)

FIG. 6 illustrates how Global Regional Offsets 23 are used todynamically transpose Seed Notes to accommodate key changes during songplay.

As the audio file/song plays, and (chromatic) Song Section markers areencountered, the current Regional Offset 23 is set to the offset valuecontained in the marker 23 in real time. This ensures that all triggerednotes will be in the proper key for what the song is currently playing.

In this example, Region #1 plays in the key of C major while Region #2is transposed to the key of F major.

FIG. 7 illustrates two selectable Instruments and their definitions.

All predefined instrument choices contain information necessary to usethat instrument in a musical composition. If the notes to be played areMIDI, each instrument 61 must be assigned to a Midi Synthesizer Patch 63within a downloadable sound set 62. The ideal Octave Range for eachinstrument is determined from the Low 64 & High 65 Octave values. InFIG. 7, the octave range for the upright bass is two octaves; the octaverange for the electric piano is five octaves.

FIG. 8 illustrates how the preliminary seed note pool 42 is expanded byapplication engine 1007 to fill an Instrument's specific Octave Range.

For each Instrument selected by the user:

The Octave Range 66 is calculated from the Low Octave 64 and High Octave65 values.

Octave Range Offsets 67 are established for each Octave contained in theOctave Range 66.

For example, all Seed Note Pools 42 are always generated for Octave #0.

If an instrument sounds best in octave #5, the offset value (inchromatic semi-tones) would be +60 (5×12 semis).

Using the Octave Range Offsets 67, one set of preliminary Seed Notes 71is generated for each octave contained in the Octave Range 66.

FIG. 9 illustrates how Offsets are dynamically applied by applicationengine 1007 to an Instrument's preliminary seed note pool to produce aplayable sympathetic note pool in real time.

All of an Instrument's play-notes start out as standard Scale Notescontained in a Scale Note Template 31. The appropriate template isindicated by chromatic analysis of the imported song. Chromatic analysisalso sets the Global Key Offset 24 (See FIG. 5).

The Global Key Offset 24 is added to the Midi Note numbers 33 of theappropriate Scale Note template 31 which transposes them into Seed Notes42 in the Detected Root Key 26 of the scanned song. (See FIG. 5).

Instrument Octave Offsets 67 are added to the Midi Note numbers 72 ofthe Preliminary Seed Notes 70. This transposes them to be within theInstrument's specified Octave Ranges 64-65.

Octave Offsets 67 are used to generate Preliminary Seed Notes 71 for alloctaves that fall within the range specified by the Low Octave 64 andHigh Octave 65 properties from the Instrument definition 61.

As the application engine 1007 plays the audio file/song in real time,and (chromatic) Song Section markers are encountered, the Global RegionOffset 84 is set to the offset value contained in the marker 23. Thisensures that all triggered notes will be in the proper key for what thesong is currently playing.

Each note in the note pool 86 has its own Index ID, 87 which is notaffected by any of the Offset values. This index establishes a way toretrieve and play the same melodic pattern across all Regions of thesong. See FIG. 10.

FIG. 10 illustrates how Melody Patterns work. Melody patterns establisha sequential list of notes (melodies) that can be retrieved from anInstrument's Sympathetic Note Pool 86 and played when a user triggersthem.

Melody patterns are predefined within software executing in theapplication engine 1007. They are required for playback and can beeither selected or established as defaults for an Instrument.

Melody Patterns can have an unlimited number of steps.

The same Melody Pattern can be used across all chromatic regions in asong by dynamically adjusting the Regional Offset 84.

Any Melody Pattern can be used with any Instrument.

To provide flexibility across all keys, Melody Patterns do not directlyreference the pattern notes by their Key designation 86. Instead, thenotes are referenced by their Index ID 87 within the Instrument'sSympathetic Note Pool (FIG. 9). This makes a Melody Pattern transparentto the Key or Octave to which it applies.

FIG. 10 shows an example of a 13 step Melody Pattern that will play“Mary Had a Little Lamb” melody in any Key and on any Instrument.

FIG. 11 illustrates the main processing flow 1100 executable byapplication engine 1007 configured to generate an interactive play-alongsong according to this disclosure.

At step 1101 the application engine 1007 receives a user selection of anaudio file/song to import the audio file/song from either a musicstreaming service or from a personal music library, and the like.

The imported audio file/song must be in a standard audio format such as.wav or .mp3, or the like. A preliminary Virtual Song map 20 isestablished showing the length of the overall song, its measurealignment grid, tempo setting markers, and where they occur in the audiofile/song as it is played. A link to the imported file 27 is saved aspart of the song definitions in definition file 1010. The length of theimported song 28 is saved as part of the song definitions.

At step 1102 the imported audio file/song is software scanned byapplication engine 1007 to determine its beat points (tempo), and itsbeat/measure alignments (time signature). The results of the scan areused by application engine 1007 to prepare a Virtual Song map 20 for theentire song by placing Tempo markers into the map at the appropriatelocations along its length, as previously discussed, as shown in FIG. 3.

At step 1103 the imported audio file is software analyzed by applicationengine 1007 to determine its overall key 26.

The audio file is also scanned along its length to determine key changesand mark where they occur.

Regional Key indicator markers 22 are placed in the virtual Song Map 20at the appropriate locations along its length.

All predesigned Key/Scale templates 31 contain a generic Scale Note pool32 in the key of C in Octave #0.

Based on the detected key for the audio file/song, a Global Key Offset24 is established. This Global offset transposes all generic Scale Notes32 from the generic key of C to Seed Notes 42 in the detected key. Forexample, if the detected key is F, the Global Key Offset 24 would be +5.

At step 1104 a basic pool of Sympathetic Seed Notes for the song isestablished by the application engine 1007 applying the Global Keyoffset to the scale notes contained in the Generic Key/Scale template.(Template Note) 32+(Global Key Offset) 24=Seed Note Pool 42.

At step 1105 the user interface/display 1004 presents the user with amenu of possible play-along instruments 61 or instrument combinations.The menu is organized in ways to simplify selections such as organizingthe instrument choices by Genre.

At step 1106 an Instrument-specific Base Pool of notes 71 is establishedby application engine 1007 for each selected instrument.

At step 1107 each Instrument selection comes with a predetermined(default) method to trigger it (Trigger Type Pulse; One-shot etc). Uponuser selection of an Instrument, the user may accept or change thetriggering characteristics.

At step 1108 each Instrument selection comes with a default selection ofplayback templates shown in FIG. 10 that the user may accept or change.Playback templates are Melodic in nature and they dictate the melodysequence of sympathetic notes being played.

At step 1109, upon completion, all song selections & properties aresaved by the application engine 1007 in an XML format that can be read,edited, and played, such as by Beamz Interactive software offered byBeam Interactive, Inc. of Scottsdale Ariz.

The appended claims set forth novel and inventive aspects of the subjectmatter described above, but the claims may also encompass additionalsubject matter not specifically recited in detail. For example, certainfeatures, elements, or aspects may be omitted from the claims if notnecessary to distinguish the novel and inventive features from what isalready known to a person having ordinary skill in the art. Features,elements, and aspects described herein may also be combined or replacedby alternative features serving the same, equivalent, or similar purposewithout departing from the scope of the invention defined by theappended claims.

1. A non-transitory computer readable medium including instructionsoperable by an electronic processor, comprising instructions for:receiving an audio file from a remote source; directly analyzing theaudio file to map in audio form and create a virtual song definitionconfigured for interactive playback; and performing chromatic analysisacross a length of the audio file to create a song section in thevirtual song definition, where each song section contains a regional keyoffset that is configured to transpose seed notes from a detected rootkey to regional specific seed notes for the song section in real-time.2. The non-transitory computer readable medium as specified in claim 1,further comprising instructions for performing tempo analysis on thelength of the audio file to create a tempo region in the virtual songdefinition, where each tempo region contains a tempo value thatestablishes a playback tempo for the region.
 3. The non-transitorycomputer readable medium as specified in claim 2, further comprisinginstructions such that when a tempo value marker is encountered duringplayback of the virtual song definition, a current tempo is set to avalue contained in the tempo value marker.
 4. The non-transitorycomputer readable medium as specified in claim 1, further includinginstructions to use pools of sympathetic notes that correspond to keychanges as the audio file plays.
 5. The non-transitory computer readablemedium as specified in claim 4, further including instructions forassociating an instrument to one said pool of sympathetic notes.
 6. Thenon-transitory computer readable medium as specified in claim 5, furtherincluding instructions for associating a plurality of instruments torespective pools of sympathetic notes.
 7. The non-transitory computerreadable medium as specified in claim 5, further comprising instructionsfor assigning a melody template to each instrument's pool of sympatheticnotes to govern how notes are retrieved and played.
 8. Thenon-transitory computer readable medium as specified in claim 1, furthercomprising instructions for saving audio file definitions in adefinition file.
 9. The non-transitory computer readable medium asspecified in claim 8, wherein definition file is an XML file.
 10. Asystem, comprising; a processor configured to: receive an audio filefrom a remote source; directly analyze the audio file to map the audiofile and create a virtual song definition map configured for interactiveplayback; and perform chromatic analysis across a length of the audiofile to create song section objects in the virtual song definition,where each song section contains a regional key offset that isconfigured to transpose seed notes from a detected root key to regionalspecific seed notes for the song section in real-time.
 11. The system asspecified in claim 10, wherein the processor is configured to: performtempo analysis on the length of the audio file to create tempo regionobjects in the virtual song definition, where each tempo region containsa tempo value that establishes a playback tempo for the region; andperform tempo analysis on the audio file to create the virtual song mapand place tempo value markers in the virtual song map to establish orchange the tempo at that point in the audio file.
 12. The system asspecified in claim 11, wherein the processor is configured such thatwhen a tempo value marker is encountered during playback of the virtualsong definition, a current tempo is set to a value contained in thetempo value marker.
 13. The system as specified in claim 10, wherein theprocessor is configured to use pools of sympathetic notes thatcorrespond to key changes as the audio file plays.
 14. The system asspecified in claim 13, wherein the processor is configured to associatean instrument to one said pool of sympathetic notes.
 15. The system asspecified in claim 14, wherein the processor is configured to associatea plurality of instruments to respective pools of sympathetic notes. 16.The system as specified in claim 14, wherein the processor is configuredto assign a melody template to each instrument's pool of sympatheticnotes to govern how notes are retrieved and played.
 17. The system asspecified in claim 10, wherein the processor is configured to save audiofile definitions in a definition file.
 18. The system as specified inclaim 17, wherein definition file is an XML file.